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Foreword 

This  special  report  describes  objectives  and  outcomes  of  ongoing  team  training  research  and 
development  (R&D)  under  the  guidance  of  the  U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences,  Rotary-Wing  Aviation  Research  Unit  (ARI-RWARU)  at  Fort  Rucker, 
Alabama.  The  Aircrew  Coordination  Training  Enhancement  (ACTE)  program  is  an  applied 
research  project  that  employs  experience,  innovation,  and  technology  to  address  the  operational 
issue,  “Can  interactive  multimedia  courseware  using  web-based  distribution  provide  the  realism 
and  relevance  necessary  for  effective  behavior-based  team  training  and  evaluation?” 

The  goal  of  the  ACT  Enhancement  effort  is  to  make  available  a  web-delivered,  interactive  aircrew 
coordination  training  system  that  provides  Army  aircrews  with  the  knowledge  and  skill-sets 
needed  to  increase  flight  safety  and  mission  effectiveness  in  daily  operations.  Focus  areas  for  the 
applied  research  were  identified  by  Army  leadership  and  the  Aircrew  Coordination  Training 
Working  Group.  They  emphasized  — 

•  Current  ACT  program  revitalization  and  enhancement  without  repeating 
previous  research 

•  Automation  and  aircraft  configuration  issues 

•  Course  length  reductions 

•  Simplified  assessment  and  evaluation  procedures 

•  Adult  learning-based  presentation,  feedback,  and  discussion 

The  products  from  the  first  phase  of  research  are  research  reports  on  the  development  and 
evaluation  of  computer-based  Aircrew  Coordination  Training.  They  include  as  attachments  two 
interactive  multimedia  courses  of  instruction  with  supporting  training,  the  ACTE  Aircrew  Course 
and  the  ACTE  Instructor  Course.  Both  courses  contain  a  fully  integrated  Data  Management 
System  that  tracks  student  demographics,  provides  graphic  feedback  displays  during  evaluation 
exercises,  and  facilitates  electronic  course  critiques.  User  testing  and  validation  results  indicate 
high  levels  of  acceptance  for  both  the  training  and  performance  evaluation  components.  Initial 
testing  of  the  prototype  courseware  on  the  Army’s  distance  learning  suite  supports  both  the  web- 
based  and  instructor  facilitated  delivery  strategies.  The  products  have  been  presented  to  the  U.S. 
Army  Aviation  Center  for  consideration  for  Army-wide  implementation. 

Initial  data  suggest  that  web-delivered,  interactive  multimedia  courseware  provides  effective 
realism  and  relevance  for  team  training  and  evaluation.  Ongoing  ARI  research  activities  include 
developing  aircraft-specific  training  support  packages  and  a  program  to  meet  the  particular  training 
needs  of  non-rated  crewmembers.  The  ACTE  serves  as  a  model  for  adult  learning-based  training 
supported  by  distance  learning  technologies. 


ZITA  M.  SIMUTIS 
Director 
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The  Aircrew  Coordination  Training  (ACT)  Challenge 


The  crew  of  the  UH-60  Black  Hawk  helicopter  was  conducting  night  training.  The  four-man  crew 
was  wearing  night  vision  goggles  in  an  extremely  dusty  environment.  CW2  Stanley  was  a  little 
nervous,  having  only  600  hours  in  this  aircraft, 
but  he  was  veiy  aware  that  his  Instructor  Pilot 
(IP),  CPT  Fulford,  had  over  3, 000  hours  of  flight 
time  and  was  highly  respected  in  the  unit  for  his 
abilities.  CPT  Fulford  was  on  the  controls  when 
they  took  off  into  a  dust  cloud.  Stanley  and  both  of 
the  crew  chiefs  sitting  in  the  back  seat  sensed  that 
the  aircraft  was  in  a  left  turn,  but  none  of  them 
said  anything,  not  wanting  to  question  CPT 
Fulford.  The  aircraft  had  indeed  been  in  a  left 
turn,  which  changed  a  stiff  right  crosswind  into  a 
tailwind  and  prevented  it  from  climbing  out  of  the 
dust  cloud.  It  contacted  the  ground,  rolled  over, 
and  was  destroyed. 


| 


The  results  of  the  Accident  Review  Board  were  conclusive.  Crew 
error,  specifically  on  the  part  of  the  instructor  pilot,  was  the  prime 
contributing  cause  of  this  accident.  CPT  Fulford  allowed  himself  to 
exceed  his  capabilities  by  attempting  to  perform  a  maneuver  that  he 
had  not  performed  in  the  aircraft  for  an  extended  period  of  time.  But 
there  was  a  deeper,  more  disturbing  element  present  here  as  well. 
Three  other  crewmembers  just  sat  there  and  let  it  occur.  All  sensed 
that  something  was  not  quite  right,  but  they  made  no  attempt  to  say 
anything  during  the  maneuver  that  may  have  prevented  the  error  that 
was  about  to  occur. 


Aircrew  Coordination  Training  Defined 

The  US  Army  defines  aircrew  coordination  as  a  set  of  principles,  attitudes,  procedures,  and 
techniques  that  transforms  individuals  into  an  effective  crew.  The  stated  objective  of  Aircrew 
Coordination  Training  (ACT)  is  to  provide  aircrews  the  knowledge,  skills  and  attitudes  necessary 
to  increase  their  mission  effectiveness,  while  decreasing  the  errors  that  lead  to  accidents. 

ACT  and  Crew/Cockpit  Resource  Management  (CRM)  programs  were  instituted  in  the  1 980’s, 
first  in  commercial  aviation  and  later  in  military  aviation,  to  address  adverse  mishap  rate  trends 
that  showed  the  inability  of  many  aviators  to  work  well  together  in  periods  of  high  stress  or 
workload  (Helmreich,  Merritt,  &  Wilhelm,  1999).  Minor  aircraft  malfunctions  were  resulting  in 
fatal  accidents  with  alarming  frequency.  While  aviators  generally  displayed  excellent  knowledge 
and  understanding  of  aircraft  systems,  operating  procedures,  rules  and  regulations  and  other 
technical  information,  they  often  displayed  a  glaring  inability  to  communicate  effectively, 
distribute  workload,  maintain  or  regain  situational  awareness  and  make  sound  decisions.  Military 
aviation  took  note  of  the  success  of  CRM  in  the  civilian  sector  and  instituted  similar  training 
programs  (Orlady  &  Foushee,  1987). 
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ACT/CRM  programs  have  been  structured  in  various  ways  and  continue  to  evolve  as  the 
perspective  changes  as  to  what  constitutes  effective  team  coordination  training.  Most  programs 
include  the  following  basic  elements: 

•  A  discussion  of  the  core  behaviors  or  basic  skill  sets  that  make  up  ACT.  Each 
program  structures  these  core  behaviors  differently,  but  all  contain  common 
elements. 

•  An  examination  of  the  applicability  of  ACT  behaviors  in  the  “real  world.”  This 
typically  takes  the  form  of  one  or  more  case  studies  of  real-world  incidents  or 
accidents  and  includes  an  analysis  of  where  or  when  proper  ACT  behaviors 
could  have  been  employed. 

•  Some  type  of  role-playing  or  practice  of  ACT  behaviors  in  a  simulated  mission 
setting,  e.g.,  line-oriented  flight  training  (LOFT)  or  its  equivalent. 

•  Some  form  of  assessment  of  the  learning  or  changes  in  attitudes  and  behaviors 
that  have  taken  place  as  a  result  of  the  training,  and  the  evaluation  of  the 
training  by  the  students. 

During  fiscal  years  1984-1989,  crew  coordination  failures  identified  by  the  Army  Safety  Center 
directly  contributed  to  the  loss  of  147  lives  and  $292  million  in  aviation  accident  costs.  In  1990, 
Army  leadership  directed  the  Army  Aviation  Center  to  focus  its  aviation  training  and  evaluation  on 
crew  performance. 

Beginning  in  1988,  the  US  Army  Research  Institute  (ART)  began  conducting  a  program  of  training 
research  that  responded  to  the  Army  need  for  better  crew  coordination  training.  This  program  of 
research  conducted  in  close  cooperation  with  the  US  Army  Aviation  Center  (USAAVNC)  in 
support  of  the  effort  to  revise  aviation  training  standards  to  incorporate  increased  emphasis  on 
crew-level  performance.  To  assist  in  the  effort  to  refocus  aircrew  training  on  the  crew  level  of 
performance,  ARI  contracted  with  Dynamics  Research  Corporation  (DRC)  for  a  two-phase 
research  effort.  First,  develop  and  test  measures  of  crew  coordination,  then  develop  and  validate  an 
Aircrew  Coordination  Training  course  using  field  aviators.  Evaluation  testing  of  aircrew 
coordination  measures  by  unit  aviators  produced  favorable  results  (Pawlik,  Simon,  Risser,  & 

Zeller,  1990).  The  US  Army  Aviation  Center  then  formed  an  Aircrew  Coordination  Working 
Group  (ACWG)  to  review  measures,  research  results,  and  companion  training  and  evaluation 
materials.  The  ACWG  revised  the  behavior-based  training  and  evaluation  designs  for  development 
and  validation  testing  of  an  aircrew  coordination  course. 

ARI  worked  closely  with  Army  aviation  training,  evaluation,  and  safety  personnel  to  develop, 
validate,  and  field  an  ACT  Exportable  Training  Package.  The  test  bed  demonstrated  and  validated 
the  program  for  training  and  evaluating  crew  coordination  skills.  Test  results  showed  that  the 
crews  performed  their  missions  significantly  more  effectively  and  safely  after  the  training  than 
before  the  training  (Simon  &  Grubb,  1993). 

The  Army  ACT  program  suite  of  methods  and  measures  included  behaviors  and  performance  (see 
Table  1).  Results  of  the  Army  ACT  validation  test  showed  significant  improvement  in  aircrews’ 
overall  mission  effectiveness  between  pre-  and  post-training  evaluation  scenarios.  During  this  test, 
all  aircrews  made  errors  in  completing  the  complex,  tactical  simulator  missions.  Pre-  to  post- 
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training  results  improved  markedly  for  navigation,  instrument  flight  recovery,  and  mission- 
threatening  error  performance  measures.  Like  the  mission  effectiveness  increases,  results  of  the 
Army  ACT  validation  test  showed  significant  improvement  in  aircrew  flying  safety  between  pre- 
and  post-training  evaluation  scenarios. 

Table  1.  1992  ACT  Validation  Test. 

Methods  and  Measures 

ACT  behaviors  or  Basic  Qualities  evaluated  with  supporting 
behavioraily  anchored  rating  scales 

Aircrew  Training  Manual  task  performance 

Mission  performance  of  two  flight  simulator  scenarios  similar  in 
difficulty  in  terms  of  time  stress,  navigational  demands,  quantity 
and  capabilities  of  simulated  threat 

Mission  Effectiveness  Results 


Overall  mission  effectiveness  +20% 


Navigation  accuracy  +38% 


Mission  objectives  +44% 


Error  Management  Results 


Course  deviations  -45% 


Arrive  at  correct  landing  zone  +38% 


No  early  descent  below  minimums  +32% 


Manage  mission  threatening  error  +27% 


Flying  Safety  Results 


Unexpected  weather  recovery  +25% 


Emergency  detection  +38% 


Number  of  aircraft  crashes  -43% 


Instructor  evaluators  rated  crew  performance  of  the  set  of  1 3  Army  ACT  behaviors  (called  Basic 
Qualities)  during  each  test  mission.  There  was  improvement  between  the  pre-training  and  post¬ 
training  evaluations  in  every  Basic  Quality  with  statistically  significant  improvements  on  12  of  13. 

The  USAAVNC  approved  the  Aircrew  Coordination  Exportable  Training  Package  (ETP)  (Pawlik, 
Simon,  Grubb,  &  Zeller,  1992)  late  in  1992,  and  formed  a  training  cadre  in  August,  1993.  This 
cadre,  made  up  of  attack,  cargo,  observation,  and  utility  subject  matter  experts  (SMEs),  was  tasked 
to  field  the  training  and  evaluation  system  to  both  the  Army  Active  and  Reserve  Components  by 
the  end  of  May  1995  (Pawlik,  Simon,  Grubb,  &  Zeller,  1993). 


Aircrew  Training  Program 

The  Army’s  Aircrew  Training  Program  (ATP)  specifies  that  aircrew  coordination  be  emphasized 
during  readiness  level  progressions  and  will  be  evaluated  during  all  evaluation  flights.  The 
inclusion  of  aircrew  coordination  in  Aircrew  Training  Manual  (ATM)  task  descriptions  reflects  the 
“crew  concept”  philosophy  that  generally  no  task  is  an  individual  undertaking.  Each  task  can  be 
performed  more  effectively  and  safely  by  the  coordinated  efforts  of  the  entire  crew.  ATM  revisions 
will  include  crew  actions  in  the  task  descriptions,  as  appropriate.  Crew  actions  define 
responsibilities,  whether  individual  or  crew,  by  describing  the  parts  of  a  task  that  an  individual  or 
group  of  crewmembers  will  perform.  Knowledge  of  the  crew  actions  for  tasks  being  performed 
will  help  crewmembers  perfonn  their  individual  actions  more  effectively  and  enhance  crew 
coordination. 

Research  and  studies  conducted  by  USAAVNC,  ARI,  and  the  US  Army  Safety  Center  (USASC) 
show  the  importance  of  good  aircrew  coordination.  A  1990  analysis  of  US  Army  aviation 
accidents  revealed  that  a  significant  percentage  of  these  accidents  resulted  from  one  or  more  crew 
coordination  errors  committed  before  or  during  the  flight.  Often  an  accident  was  the  result  of  a 
sequence  of  undetected  crew  errors  that  combined  to  produce  a  catastrophic  result.  Additional 
research  by  ARI  showed  that  even  when  accidents  are  avoided,  these  same  errors  can  result  in 
degraded  performance.  A  systematic  analysis  of  these  error  patterns  identified  specific  areas  where 
crew-level  training  could  reduce  the  occurrence  of  such  errors  and  break  the  error  chains  leading  to 
accidents  and  poor  performance  (Peusch  &  Hicks,  2001). 

Broadly  defined,  aircrew  coordination  is  the  interaction  of  crewmembers  necessary  for  the  safe, 
efficient,  and  effective  performance  of  tasks.  Working  with  this  definition,  the  USAAVNC  and 
ARI  translated  crew  coordination  concepts  into  a  set  of  5  crew  coordination  objectives  and  13 
Basic  Qualities  (see  Appendix  A).  Crew  coordination  objectives  are  the  on-going  crew  mission 
responsibilities  that  form  the  organizing  structure  for  aircrew  coordination  training  and  evaluation 
behaviors.  Basic  Qualities  are  the  behaviors  and  skills  that  can  be  trained,  observed,  and  evaluated 
relative  to  crew  coordination  performance.  Both  Crew  Coordination  Objectives  and  Basic 
Qualities  have  been  incorporated  into  the  USAAVNC  Aircrew  Coordination  ETP.  Each  basic 
quality  is  defined  in  terms  of  observable  behaviors  that  represent  superior,  satisfactory,  or 
unsatisfactory  levels  of  crew  coordination.  These  Basic  Qualities  and  goals  are  summarized  in 
each  aircraft  ATM.  The  Aircrew  Coordination  ETP  contains  detailed  guidance  and  performance 
descriptions  for  use  by  commanders  to  evaluate  ACT  performance  and  qualify  crewmembers. 

Description  of  the  Initial  Army  ACT  Program 

The  U.S.  Army  implemented  its  version  of  ACT  in  1994  (Department  of  the  Army,  1992).  As  a 
result  of  the  initial  ACT  training  program,  Army  aircrews  learned  behavioral  skills  and  team 
coordination  techniques  that  helped  them  to  remain  focused  and  ready  to  deal  with  emergencies 
and  unforeseen  problems  so  they  were  able  to  better  concentrate  on  mission  objectives. 
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Implementation 

Crew  coordination  training  provides  the  knowledge,  skills,  and  attitudes  to  aircrews  that  increase 
their  mission  effectiveness,  while  decreasing  the  cockpit  errors  that  contribute  to  accidents. 
Following  the  implementation  of  the  initial  Army  ACT  in  the  mid-nineties,  the  Class  A  accident 
rate  dramatically  dropped  (see  Figure  1).  Army  ACT  was  presented  as  “one-time  training”  without 
annual  continuation  or  sustainment  training.  Though  other  variables  may  have  contributed  to  the 
decrease,  the  Class  A  aviation  accident  rate  increased  when  ACT  was  no  longer  emphasized. 


Figure  1.  Impact  on  accident  rate. 


Commanders  and  aircrews  alike  acknowledged  the  benefit  of  the  mandatory,  one-time  training  that 
was  received  by  all  aviators  within  the  Army  aviation  community.  The  initial  program  did  not 
address  sustainment  issues  and  did  not  package  the  training  in  a  program  that  would  facilitate  such 
training.  Therefore,  sufficient  funds  were  not  provided  for  developing  a  program  to  sustain  this 
important  training.  Funding  issues  notwithstanding,  significant  personnel  turbulence  associated 
with  downsizing  the  force  since  the  1994  program  inception  has  potentially  resulted  in  an  erosion 
of  the  safety  gains  initially  realized  as  a  result  of  ACT.  Finally,  the  atrophy  of  skills  and  the 
decline  of  experience  levels  that  occurred  during  successive  years  of  limited  defense  funding 
coincided  with  a  sharp  increase  in  accident  and  incident  rates  (Allman,  2002). 

Lack  of  effective  aircrew  coordination  continues  to  be  cited  as  a  definite  or  suspected  contributing 
factor  in  aviation  flight  accidents,  and  it  is  a  factor  limiting  attainment  of  the  full  mission 
effectiveness  of  Army  aviation.  For  example,  the  Director  of  Army  Safety  reported  in  the 
December  1999  issue  of  Flight  fax,  “In  fact,  FY99  produced  Army  aviation’s  worst  safety 
performance  since  Desert  Shield/Desert  Storm.”  The  ACT  program  has  not  been  updated  since  its 
original  introduction.  Currently,  ACT  is  conducted  in  the  classroom  via  eight  hours  of  instruction, 
with  no  mandatory  follow  on  training.  Instructors  responsible  for  evaluating  and  reinforcing  this 
academic  training  receive  four  hours  of  academic  training  with  no  exam  to  determine  competency. 
Temporary  measures  such  as  awareness  videos,  assistance  visits,  safety  newsletter  articles,  and  a 
web-based  training  support  package  have  been  ineffective  substitutes  for  focused  ACT  training. 
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Enhancing  Behavior-based  Training  and  Evaluation 


Aviation  Safety  Investment  Strategy  Team  (ASIST) 

Lack  of  effective  aircrew  coordination  continues  to  be  cited  as  a  definite  or  suspected  contributing 
factor  in  aviation  flight  accidents,  and  it  is  a  factor  limiting  attainment  of  the  full  mission 
effectiveness  of  Army  aviation.  The  Aviation  Safety  Investment  Strategy  Team  (see  Figure  2)  was 
chartered  in  1994  to  define  measurable  accident  prevention  goals  and  identify  the  most  important 
Army-wide  investments  needed  to  achieve  them.  The  ASIST  study  reported  that  a  crew 
coordination  sustainment-training  program  would  help  attain  the  Army  accident  reduction  goals  at 
the  least  investment  cost  (Hicks  &  Peusch,  2000). 

Figure  2.  Aviation  Safety  Investment  Strategy  Team  (ASIST). 


>  Personnel  Turbulence  and  Operations  Tempo 

>  Increasing  Reliance  on  Active-Reserve  Unit  Mix 


>  Aviation  Safety  Investment  Strategy  Team  (ASIST) 

•  In-depth  analysis  of  accident  experience 

•  All  force-modernized  aircraft  during  FY  94-98 

•  Eight  of  top  ten  hazards  (290  total  hazards)  are  crew 
coordination  related 

•  Number  four  of  top  ten  controls  (249  total  controls)  is 
crew  coordination  related 

Develop,  monitor,  and  evaluate  a  Crew  Coordination  Sustainment  Training 
program  Integrated  Into  aviation  tasks  (Control  Number  4  In  “One  to  Goal  List”) 


Aircrew  Coordination  Training  Master  Plan 

The  ACTE  program  is  being  developed  under  the  guidelines  established  by  the  ARI-produced 
Army  Aircrew  Coordination  Training  Master  Plan.  The  plan  provides  a  proactive,  multi-phased 
course  of  continuous  improvement  to  maximize  Army  aviation  modernization  investments  and 
complement  leadership  training  initiatives  (see  Figure  3).  The  now  completed  initial  research  task 
was  to  establish  a  logical  master  plan  that  provides  a  business  process  for  achieving  the  desired 
ACTE  program  end-state.  Aircrew  training  programs  currently  provided  by  Army,  other 
Department  of  Defense  (DoD)  agencies,  and  commercial  air  carriers  were  reviewed  as  benchmarks 
for  the  training  and  evaluation  of  aircrew  and  team  coordination  in  both  the  institution  and  unit 
environments.  Issues  relevant  to  both  rated  and  non-rated  aircrew  members  in  rotary  wing  and 
fixed  wing  environments  were  identified.  Insights  and  operational  trend  data  provided  by  the 
aviation  safety,  standards  and  evaluation,  and  training  communities  were  examined. 
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Figure  3.  Aircrew  Coordination  Training  Master  Plan. 


Begin  Divestment  of  Legacy 
Systems  (UH-1.AH-1) 


Begin  Flight  School  XXI 
Begin  Aviation  Combined  Arms 
Tactical  Trainer  Deployment 
Begin  Distribution  of  Trans¬ 
formation  Aircraft  to  RC 


>  Fully  Implement  Flight  School  XXI 
V  Begin  UH-60M  Production 

>  204  Distance  Learning  Sites 
Operational 


ACTE  Master  Plan 


Divest  Legacy/Transitional  Systems 
Convert  All  Corp  and  Division 
Brigade  (AC-RC)  to  Transitional 
Force  Structure 

Aviation  Combined  Arms  Tactical 
Trainer  Operational 


ACTE  Phase  1 

ACTE  Phase 

•  Establish  Interim  Aircrew  Coordination  Training 

•  Establish  Permanent  Aircrew  Coordination 

Working  Group 

Training  Working  Group 

•  Review  Current  Aircrew  Coordination  Training 

•  Develop  Separate  Aircrew  Coordination  Training 

Programs 

Policy 

•  Develop  Master  Plan 

•  Develop  Refresher  Training  Support  Packages 

•  Include  Aircrew  Coordination  Training  in  Aviation 

•  Train  Unit  Leaders  and  Instructors 

Leader  Conferences 

•  Incorporate  Aircrew  Coordination  Training 

•  Recommend  as  Senior  Readiness  Oversight 

Evaluation  into  Readiness  Level  and  Aviation 

Council  Interest  Item 

Proficiency  and  Readiness  Test 

•  Develop  Aircrew  Coordination  Training 

•  Integrate  Aircrew  Coordination  Training  into  Flight 

Performance  Evaluation  System 

School  XXI 

•  Develop  Core  Training  Modules 

•  Develop  Aircrew  Coordination  Training  Website 

•  Pre-test,  Dem-Val,  and  Field  Test  Courses 

•  Develop  Accident  Investigative  Tool 

•  Integrate  Aircrew  Coordination  Training  into 

Advanced  Aircraft  Simulators  and  Distributive 

Interactive  Simulation  Scenarios 

•  Provide  Non-rated  Crewmembers  Training 

•  Develop  Evaluation-based  Feedback  System 

•  Provide  Site  to  Host  ACTE  Courseware 

CTE  Phase  III 

Integrate  to  include  all  aircrew  members 
Implement  Accident  Investigative  Tool 
Train  Accident  Investigators 
Develop  Advanced  Aircrew  Coordination 
Training  Scenarios 

Develop  Aircrew  Coordination  Training  Website 
(Resources  Repository) 

Establish  O&M  Recurring  Funding  Plan 
Establish  Team  Training  Research  and 
Development  Program 


Policy 


Training 


Training 


Evaluation/Risk  Mgmt 


Evaluation/Risk  Mgmt 


Aircrew  Coordination  Training  Enhancement  (ACTE)  Research 

The  objective  of  the  research  effort  to  enhance  Army  ACT  is  to  improve  the  crew  and  team 
coordination  effectiveness  of  Army  aircrews  in  their  day-to-day  mission  planning  and  flight 
operations.  The  enhanced  ACT  program  builds  on  the  original  exportable  training  package, 
revitalizing  it  from  a  one-time  training  event  and  enhancing  it  to  a  dynamic,  relevant  program  that 
is  continuously  updated  and  improved  (see  Figure  4).  Establishing  and  maintaining  a  unit-level 
command  climate  that  promotes  the  use  of  team  coordination  behaviors  will  realize  this  objective 
and  places  equal  emphasis  on  technical  and  team  coordination  skills  in  daily  flight  operations. 
Instructor  pilots  and  ACT  facilitators  in  aviation  units  are  key  to  the  institutionalization  of  a 
successful  ACT  program. 

ARI’s  Rotary-Wing  Aviation  Research  Unit  convened  an  Aircrew  Coordination  Working  Group 
(ACWG)  with  recognized  knowledge  and  experience  in  aircrew  coordination  training,  standards 
and  evaluation,  safety,  and  human  factors  to  review  measures,  methods,  and  training  materials 
prior  to  inclusion  in  the  prototype  ACTE  courses.  The  ACWG  is  composed  of  designated 
representatives  from  the  USAAVNC,  the  USASC,  and  the  Army  National  Guard.  Other  agencies 
that  have  information  or  resources  that  contribute  to  planning  or  developing  the  program  or  have  a 
role  in  implementing  or  evaluating  the  program  are  included. 
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Figure  4.  Old  versus  enhanced  ACT. 


OLD  Revitalized 


Initial,  one-time  qualification  training 
Facilitated  discussion  of  paper-based 
courseware  with  generic  video  segments 
Basic  qualities  of  ACT  behaviors 
Research-based  evaluation  methods 
and  measures 


Sustainment  training  with  annual  updates 
Advanced  adult  learning  technologies  in 
electronic  classroom  environment 
ACT  skills  level  of  instruction 
Simplified  evaluation  tools  and  techniques 
Reduced  course  length 
Reduce  instructor  requirements  for  units  to 
implement  Tailored  to  aircraft  and  unit 
missions 

Hands-on  applications  based  on  current 
operational  situations 
Enhanced  error  management  tools  and 
techniques 

Unit-level  training  packages  with  web- 
based  feedback  and  support 
Ongoing  program  evaluation  and 
management,  i.e.,  training,  safety, 
standards  and  evaluation 


The  current  enhancement  program  managed  by  the  ARI  is  a  multi-year,  multiphase  program  of 
applied  research  structured  in  three  major  phases  -  upgrade  and  sustain  the  existing  ACT  program, 
refresh  and  maintain  the  upgraded  ACT  program,  and  deploy  advanced  ACT  applications.  Phase  I 
of  the  research  effort  to  upgrade  and  sustain  the  current  ACT  program  is  completed.  Two  ACTE 
prototype  courseware  programs  have  been  delivered  to  the  Army,  the  Aircrew  Course  and  the 
Instructor  Course.  Each  course  of  instruction  is  the  integration  and  end  product  of: 


•  Analysis  of  the  current  aircrew  coordination  training  program 

•  Definition  of  evaluation  tools  and  techniques  for  assessing  along  specific 
behavioral  proficiency  dimensions  overall  crewmember  team  performance 

•  Development  of  prototype  focused  interventions  for  training  and  evaluating 
team  coordination  behaviors  and  for  managing  risk 

•  Validation  of  prototype  team  training  and  evaluation  techniques  in  selected 
aviation  units 

•  Field-testing  of  prototype  training,  evaluation,  and  technology  products 
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Enhancement  Prototype  Products 

When  the  Phase  I  prototype  products  are  fully  developed  and  deployed,  the  Aircrew  Coordination 
Training  Enhancement  (ACTE)  program  will  provide  mission-oriented  sustainment  training  and 
provision  for  web-accessible  updates. 

Simplified  Performance  Evaluation  System 

The  products  necessary  to  provide  a  simplified  performance  evaluation  system  consisted  of  a 
practical  assessment  methodology  and  a  suite  of  quantitative,  field-usable  measures  to  allow 
across-platform  and  across-crew  configurations  evaluation  of  ACT  behaviors  and  skills. 
Implementing  the  evaluation  system  required  a  tool  for  recording  ACT  performance  evaluations  to 
support  facilitation  of  team  performance  improvement  during  the  after-action  review. 

The  central  product  is  a  set  of  observable  measures  of  individual  and  collective  behavior,  the 
Behaviorally  Anchored  Rating  System  (BARS).  The  BARS  is  gleaned  from  extensive  research 
across  DoD  and  commercial  aviation  communities  (e.g.,  Leedom  &  Simon,  1993;  Simon,  1990) 
and  served  as  a  precursor  to  courseware  content  design  and  development.  The  BARS  evaluation 
criteria  are  based  on  issues  relevant  to  both  rated  and  non-rated  aircrew  members  in  rotary  wing 
and  fixed  wing  environments  and  serve  as  the  benchmarks  against  which  crew  team  behaviors  are 
evaluated. 

The  measurement  of  aircrew  coordination  behavior  is  a  critical  component  of  the  aircrew 
coordination  program  and  is  central  to  the  training  content  design  and  delivery.  The  vehicle  for 
documenting  these  evaluations  is  the  ACT  Performance  Evaluation  Checklist  (see  Appendix  B) 
which  is  based  on  the  5  Crew  Coordination  Objectives  (CCO)  and  13  Basic  Qualities  (BQ) 
accepted  by  the  Army  as  descriptors  of  aircrew  coordination  behavior.  ACT  behaviors  and  skills 
are  organized  by  CCO  and  are  rated  using  a  seven-point  scale  with  values  ranging  from  1,  Below 
Standards  to  7,  Exceeds  Standards  (see  Figure  5).  In  an  electronic  classroom  setting,  learners 
observe  a  training  vignette  and  are  given  the  opportunity  to  evaluate  aircrew  behaviors  and  skills 
using  an  on-line  electronic  ACT  Performance  Evaluation  Checklist.  Their  individual  responses  are 
aggregated  as  a  histogram  and  displayed  to  the  class  to  serve  as  the  basis  for  instructor-facilitated 
discussion  to  achieve  learning  objectives. 


Figure  5.  BARS  numeric  rating  scale. 
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1 
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3 

4 
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Written  descriptions  are  provided  for  the  ACT  behaviors  and  skills  and  levels  of  performance  for 
rating  aircrews  at  the  values  of  1, 4,  and  7.  These  descriptions  serve  as  behavioral  “anchors”  and 
are  designed  to  assist  in  determining  how  well  an  aircrew  performs  ACT  behaviors  and  skills  in 
relation  to  a  well-defined  set  of  performance  criteria.  The  anchors  are  used  as  the  standard  for 
evaluating  ACT  performance.  This  avoids  the  trap  of  norm  referencing,  i.e.,  comparing  one 
aircrew’s  performance  with  that  of  another.  An  aircrew’s  performance  is  always  rated  solely  in 
relation  to  the  anchors.  This  has  long-term  implications  for  the  objective  measurement  of  aircrew 
coordination  improvement. 
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Web-based  Courseware  Development 

ACTE  courses  are  designed  for  access  via  an  Internet  browser.  Courseware  web  application 
components  are  compliant  with  Army  Distance  Learning  software  requirements  and  graphics  are 
produced  with  a  suite  of  Macromedia  Flash  and  Dreamweaver  UltraDev  authoring  tools.  A 
Microsoft  Access  hosted  Data  Management  System  (DMS)  that  is  fully  compatible  with  emerging 
learning  management  system  (LMS)  standards  is  used  to  manage  course  administration  tasks,  track 
student  demographics,  and  collect  student  critique  data.  A  key  feature  of  the  DMS  is  its  ability  to 
capture  student  responses  from  their  use  of  the  ACT  Performance  Evaluation  Checklist  and 
provide  graphic  feedback  displays  of  practical  exercise  results.  Student  reports  can  be  printed 
directly  from  a  web  browser.  Critique  items  were  developed  to  collect  participant  data  on  the 
following  information  areas:  Course  structure  and  delivery,  length  of  lesson,  instructor,  equipment, 
and  course  effectiveness.  A  five-point  response  scale  with  values  ranging  from  1  (Strongly 
Disagree),  3  (Neither  Agree  nor  Disagree),  and  5  (Strongly  Agree)  were  presented  for  each  item  in 
the  electronic  course  critique. 

Aircrew  Course 

The  Aircrew  Course  consists  of  five  modules  of  instruction:  Introduction,  an  ACT  Principles  and 
Risk  Management  Review  module,  a  Case  Study,  a  Problem  Solving  Exercise,  and  a  Conclusion 
(see  Figure  6). 


Figure  6.  Aircrew  Course  design. 


Course  Map  -  ACT  Aircrew  Course 
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1  Case  Study 
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1  Conclusion 
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•  Contains  5  Modules  with  Mutliple  Lessons 

•  Modules  1 ,  2  and  5  Web-Based,  Self  Paced 

•  Modules  3  and  4  Instructor  Facilitated 

•  One  day  in  length,  Distance  Learning  Deliverable 


Both  the  Principles  Review  and  Risk  Management  modules  employ  extensive  use  of  narrated 
audio  video  files.  The  narration  guides  the  user  through  the  course  material  while  the  audio  video 
files  provide  support  at  the  exact  time  of  cognitive  need.  Comprehension  checks  provide  learners 
the  opportunity  to  check  their  retention  and  feedback  relating  to  their  understanding  of  important 
information.  A  linked  list  of  related  topics  is  provided  on  each  topic  header  page  for  those  aircrew 
members  who  desire  a  more  in-depth  review. 
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The  case  study  module  provides  an  opportunity  for  aircrew  members  to  become  familiar  with  the 
performance  evaluation  system  and  the  BARS  for  their  use  in  promoting  and  evaluating  team 
coordination  behaviors.  Once  they  have  reviewed  the  CCOs  and  BQs,  students  are  presented  a 
current  case  study.  A  central  feature  of  the  case  study  is  an  audio  video  vignette,  based  on  a 
mishap  taken  from  actual  USASC  files,  that  focuses  on  automation  or  crew  configuration  related 
accident  or  incident  trends.  Once  students  observe  the  vignette,  they  are  given  the  opportunity  to 
evaluate  aircrew  behaviors  using  the  on-line  electronic  ACT  Performance  Evaluation  Checklist. 
Their  individual  responses  are  aggregated  as  a  histogram  and  displayed  to  the  class  to  serve  as  the 
basis  for  instructor-facilitated  discussion  to  achieve  learning  objectives.  Finally,  the  learners  are 
presented  a  problem  solving  exercise  in  which  they  are  given  the  opportunity  to  apply  their  ACT 
knowledge  and  evaluation  skills  to  a  simulation  derived  from  actual  events  and  experiences  in  the 
field. 

Instructor  Course 

The  Instructor  Course  consists  of  six  modules  of  instruction:  Introduction,  ACT  Instructional 
Setting  module,  Evaluation  Tools  and  Techniques,  Observation  and  Evaluation  Exercise,  a  module 
on  Facilitation  Skills,  and  a  Conclusion  (see  Figure  7). 

Figure  7.  Instructor  Course  Design. 
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•  Contains  6  Modules  with  Multiple  Lessons 

•  Modules  2  through  5  Instructor  Facilitated 

•  One  day  in  length,  Distance  Learning  Deliverable 


Completion  of  the  ACTE  Aircrew  Course  by  unit  instructor  pilots  is  a  prerequisite  to  attending  the 
Instructor  Course.  Instructor  pilots  must  be  highly  proficient  in  all  ACT  behaviors  and  skills 
learned  during  the  Aircrew  Course  and  they  must  be  able  to  apply  and  evaluate  them  in  their 
organization’s  mission  environment.  As  the  role  model  of  ACT  behaviors  and  skills  for  other 
aircrew  members,  they  must  “walk  the  walk”  every  flight  and  every  simulation  period.  The  ACT 
Instructional  Setting  module  emphasizes  the  ACT  Instructor’s  role  in  an  operational  unit,  relates 
the  ACT  Crew  Coordination  Objectives  (CCOs)  and  Basic  Qualities  (BQs)  to  the  Army  Risk 
Management  Process,  and  demonstrates  their  use  as  control  measures  to  mitigate  risk. 
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Presentation  approaches  and  features  similar  to  those  used  in  the  Aircrew  Course  provide  in-depth 
instruction  and  practice  exercises  on  applying  the  ACT  Performance  Evaluation  System  tools  and 
techniques  to  achieve  reliable  ratings  of  observed  aircrew  situations.  The  Instructor  Course 
provides  a  module  consisting  of  two  separate  lessons  on  detailed  evaluation  tools  and  evaluation 
techniques  to  equip  unit  instructor  pilots  as  experts  in  identifying,  evaluating,  and  facilitating  ACT 
performance  improvement.  A  second  module  devoted  to  practice  exercises  provides  three  mission 
related  situations  for  instructors  to  improve  their  use  of  the  performance  evaluation  system  and 
establish  a  pattern  of  rating  reliability. 

Inherent  to  the  ACT  performance  evaluation  process  is  the  ability  to  identify  and  apply  the  CCOs 
and  BQs  to  operational  and  simulated  mission  settings.  Courseware  support  materials  for  the 
Instructor  Course  include  an  Instructor  Guide  and  ACT  event  driven  scenario  outlines  for  use  in 
simulator  or  aircraft  evaluations  of  aircrew  ACT  performance. 

Additionally,  to  ensure  that  the  ACT  lessons  learned  during  unit  operational  or  training  missions 
do  not  go  unheeded,  students  will  be  taught  the  facilitation  skills  necessary  to  ensure  that  unit 
aircrews  do,  in  fact,  identify,  apply  and  assimilate  such  lessons  learned  into  future  missions  on  a 
daily  basis.  This  training  requires  well-developed  observational  and  evaluative  skills.  The  use  of 
vignettes  and  the  electronic  Performance  Evaluation  Checklist  supports  practice  exercises  in  the 
Facilitation  Skills  module. 

Distance  Learning  Delivery 

Both  the  Aircrew  Course  and  Instructor  course  are  accessible  on  the  World  Wide  Web.  A  number 
of  distribution  tests,  or  evaluations  of  the  courseware  deliverability,  were  conducted  throughout  the 
testing  and  pre-fielding  periods  of  Phase  I.  In  each  test,  the  courseware  ran  without  significant 
error  and  the  few  technical  issues  presented  were  easily  remedied  with  minor  technical  adjustments 
to  the  underlying  programming.  Importantly,  the  distribution  tests  confirmed  the  utility  of  the 
ACTE  courseware  for  simultaneous  consumption  by  geographically  dispersed  audiences. 
Additionally,  distribution  of  the  courseware  in  both  a  private  local  area  network  (LAN)  and  CD 
ROM  format  were  tested  and  confirmed  (see  Figure  8).  The  outcome  of  the  testing  of  courseware 
components  ensures  that  the  Army  aviation  community  can  make  ACTE  available  to  aviators 
worldwide. 


Figure  8.  Local  Area  Network  and  Classroom  XXI  Delivery 


12 


Performance  and  Effectiveness  Results 

After  the  board findings  were  released,  the  unit  sat 
down  and  conducted  a  sincere  self-assessment  of 
their  crew  coordination  performance.  They  were  all 
qualified  in  aircrew  coordination  training,  either 
from  the  original  team  training  in  1994-95,  Initial 
Entry  Rotaiy  Wing  training,  or  the  Exportable 
Training  Package.  What  had  happened  to  their 
skills?  What  could  they  do  to  regain  or  enhance 
these  skills?  By  chance,  ART  was  bringing  a  new 
ACT  enhancement  prototype  program  to  their 
installation  for  usability  assessment.  Could  this 
enhanced  program  help  to  revive  their  behaviors 
and  skills  that  they  once  had?  They  had  little  to  lose 
by  participating  in  the  program.  In  just  under  two  months  they  were  due  to  deploy  to  the  National 
Training  Center  (NTC),  and  there  they  would  encounter  the  same  type  of  environmental  conditions 
that  created  the  situation  for  the  accident.  The  unit  decided  to  sign  on  for  the  evaluation  of  the 
research  program,  and  was  able  to  refresh  24  of  their  crewmembers  in  ACT.  At  the  conclusion  of 
this  training,  all  participants  felt  that  their  skills  were  enhanced,  and  were  ready  to  apply  these 
skills  on  the  NTC  rotation. 

The  effectiveness  of  the  ACTE  training  and  evaluation  components  was  measured  at  increasing 
levels  of  fidelity  and  scope.  The  ACWG  actively  participated  in  the  courseware  design  and 
development  process,  approved  release  of  prototype  products  for  unit  testing,  and  reviewed  test 
results  and  approved  subsequent  products  as  they  progressed  through  the  prototype  courseware 
development  and  testing  process.  The  sequence  of  prototype  product  design,  development,  review, 
approval,  testing,  improvement  began  with  user  pretest  of  core  modules,  continued  through 
usability  assessment  and  field  study,  and  concluded  with  operational  field  testing  of  the  final 
prototype  products.  The  demonstration  and  validation  of  the  effectiveness  of  the  Aircrew  and 
Instructor  Courses  occurred  as  a  two-part  process  consisting  of  a  usability  assessment  and  a  field 
study  followed  by  a  final  operational  field  test. 

Usability  Assessment 

For  the  usability  assessment,  use  of  the  courseware  was  observed  and  assessed  in  selected  Army 
National  Guard  (ARNG)  aviation  units  that  represented  cargo  and  observation  aircraft,  missions, 
and  operating  conditions.  A  type  of  formative  evaluation,  the  usability  assessment  employed  the 
prototype  courseware  in  a  field  setting  using  representative  groups  of  the  target  training  audience. 
Data  were  gathered  on  Aircrew  and  Instructor  course  usability,  effectiveness,  and  areas  of  needed 
interface  and  content  improvement. 

Using  unit  classrooms  and  facilities,  contractor  instructors  presented  the  Aircrew  and  Instructor 
Courses  in  two  four-hour  periods  each  using  the  contractor  provided  LAN  of  notebook  computers. 
ARI  reviewed  and  approved  the  automated  data  collection  items.  Units  identified  and  scheduled  20 
participants  to  provide  a  cross  section  of  highly  experienced  aircrew  members  and  instructors. 
Participant  ratings  for  the  Aircrew  Course  averaged  4.0  or  higher  on  a  5-point  scale  (see  Table  2). 
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Table  2.  Usability  Assessment  Participant  Ratings. 


Rating  Items 

Aircrew 

Course 

Instructor 

Course 

Courseware  structure  and  navigation  logical  and 
understandable 

4.0 

3.8 

Comprehension  checks  understandable 

4.1 

Amount  of  information,  pace  and  time  good 

(On  5-point  scale,  1=  “too  little”,  5=“too  much”;  3  is  optimal) 

3.0 

3.1 

Vignettes  allowed  observation  &  related  ACT  to  risk 
management 

4.0 

3.8 

Instructor  knowledge,  preparation,  and  clarity 

4.6 

4.4 

Student  workstation  equipment 

4.6 

Performance  Evaluation  System  and  Aircrew  Guide  helpful 

4.2 

Content  relevant 

4.3 

Positive  effect  on  flying  safety 

4.4 

4.1 

Effect  as  refresher  to  previous  Aircrew  Coordination  Training 

4.2 

Positive  effect  on  mission  effectiveness 

4.0 

Lessons  helped  understand  how  to  identify  ACT  behaviors 

4.1 

Lessons  helped  understand  how  to  evaluate  ACT 
performance 

4.0 

Lessons  helped  understand  how  to  facilitate  ACT  performance 
improvement 

4.1 

Instructor  Evaluator  Guide  helpful  for  implementing  ACT  in 
daily  ops 

4.3 

The  subsequent  analysis  of  usability  assessment  ratings  and  comments  indicated  wide  and 
enthusiastic  acceptance  of  the  courseware  as  necessary  and  beneficial  to  Army  aviation  flight 
safety.  Example  participant  comments  include: 

•  “It  was  quick  and  to  the  point  and  covered  applicable  material  without  dragging 
on  for  a  week  or  two.”  (Aircrew  Course) 

•  “I  enjoyed  the  interaction  of  the  problem  solving  exercises.  I  think  this  is  where 
the  real  learning  takes  place.”  (Aircrew  Course) 

•  “New  information  and  examples  should  be  added  at  least  annually  for 
continuation  training  to  be  effective.”  (Aircrew  Course) 

•  "The  interaction  of  other  Instructor  Pilots  on  an  important  topic  that  can  model 
the  attitude  of  a  unit  by  helping  mold  the  attitudes  of  individuals  in  the  unit  and 
make  them  function  as  a  team.”  (Instructor  Course) 
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Field  Study 

The  field  study  was  designed  to  compare  units  immediately  before  and  after  receiving  the 
prototype  ACTE  training  and  evaluation  instruction.  Field  study  participants  were  aviation  units 
that  represented  utility  aircraft,  missions,  and  operating  conditions.  The  study  provided  evidence  of 
relative  improvement  due  to  training.  It  also  supported  descriptive,  graphical,  and  user  comments 
analysis  (see  Table  3). 


Table  3.  Field  Study  Measures. 


Measurement  Area 

Measurement  Instruments 

ACT  Behaviors 

Performance  Evaluation  Checklist 

Behaviorally  Anchored  Rating  System  (BARS) 

ATM  Task  Performance 

Scenario  Worksheets 

Mission  Effectiveness 

Scenario  Worksheets 

Crew  Related  Errors 

Scenario  Worksheets 

Course  Critique 

Data  Management  System  Scalar  Critique  Items 

Data  Management  System  Open-ended  Items 

Prototype  courseware  for  the  Aircrew  and  Instructor  Courses  was  used  to  train  observer- 
evaluators,  establish  inter-rater  reliability,  and  standardize  the  field  study  scenarios.  Once 
observer-evaluators  were  trained,  a  total  of  35  unit  leaders,  instructor  pilots,  and  aircrews 
participated  in  structured  training  and  evaluation  events  (see  Figure  9). 


Figure  9.  Field  study  events. 


/^^  Key  Personnel  /^Observer  Evaluator^^ 

Advance  Site  Visit  Team  Training  Classes 

V  11  Oct  01  J  \  29-31  Oct  01  J 

/  Aircrew  Pre-Training  \ 

Evaluation  Missions  { 

\  5-6  Nov  01  J 

J 

L 

f  Key  Personnel  \  f  Aircrew  Post-Training  \ 

(  Out  Brief  Evaluation  Missions 

V  15  Nov  01  J  \  13-14  Nov  01  J 

f  Aircrew  ACT  \ 

l-^“{  Enhancement  Training  ) 

V  7-8  Nov  01  J 

As  in  the  usability  assessment,  field  study  data  were  analyzed  and  the  findings  reported  to  the 
Aircrew  Coordination  Working  Group.  Participant  ratings  for  the  Aircrew  Course  averaged  4.0  or 
higher  on  a  5-point  scale  (see  Table  4).  Results  of  the  simulator-based  evaluation  scenarios 
indicated  measurable  improvement  between  pre  and  post  training  mission  effectiveness  and  flying 
safety,  despite  the  fact  that  all  participants  have  been  previously  ACT  qualified  (see  Table  5). 
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Table  4.  Field  Study  Participant  Ratings. 


Rating  Items 

Aircrew 

Course 

Instructor 

Course 

Tutorial  provided  knowledge  to  navigate  courseware 

4.3 

Courseware  structure  and  navigation  logical  and 
understandable 

4.6 

4.0 

Comprehension  checks  understandable 

4.2 

Amount  of  information,  pace  and  time  good 

(On  5-point  scale,  1=  “too  little”,  5- ‘too  much";  3  is  optimal) 

3.0 

3.0 

Vignettes  allowed  observation  &  related  ACT  to  risk 
management 

4.1 

4.0 

Instructor  knowledge,  preparation,  and  clarity 

4.8 

CO 

Student  workstation  equipment 

4.9 

Performance  Evaluation  System  and  Aircrew  Guide  helpful 

4.3 

Content  relevant 

4.4 

Positive  effect  on  flying  safety 

4.4 

4.0 

Effect  as  refresher  to  previous  Aircrew  Coordination  Training 

4.6 

Positive  effect  on  mission  effectiveness 

4.3 

Lessons  helped  understand  how  to  identify  ACT  behaviors 

4.2 

Lessons  helped  understand  how  to  evaluate  ACT 
performance 

4.2 

Lessons  helped  understand  how  to  facilitate  ACT  performance 
improvement 

3.8 

Instructor  Evaluator  Guide  helpful  for  implementing  ACT  in 
daily  ops 

4.3 

Example  field  study  participant  comments  include: 

•  “The  actual  interaction  between  instructor  and  student;  being  evaluated  and 
seeing  the  relevance.”  (Aircrew  Course) 

•  “The  multi-media  usage  was  a  very  good  combination  of  resources  and  kept  me 
interested  throughout  the  entire  lesson.  It  was  very  easy  to  navigate.  The 
scenarios  and  role-play  were  very  helpful.”  (Aircrew  Course) 

•  “It  provides  a  solid  foundation  for  all  ACT  Instructors  to  work  from.  It  should 
standardize  how  ACT  is  trained  and  evaluated  throughout  the  Army.” 
(Instructor  Course) 
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Table  5.  ACT  Versus  ACTE  Pre-Post  Training  Effectiveness. 


Comparison  Points 

ACT  (1992) 

ACTE  (2001) 

Validation  Test 

16  UH-60  aircrews 

18  hours  classroom 
academics  2  simulator  training 
missions 

8  UH-60  aircrews 

8  hours  web-based 

CBT  Courseware 

Measures  and  Metrics 

1993  final  report 
Descriptive  statistics  and 
significance  test  data 

2002  final  report 
Descriptive  statistics 

ACT  Behaviors 

+34%  (13  Basic  Qualities) 

+14%  (5  Crew  Objectives) 

ATM  Tasks 

+33% 

+11% 

Mission  Effectiveness 

+20% 

+20% 

Crew  Related  Errors 

-  Manage  Threatening  Errors 

+27% 

+37% 

-  Number  of  aircraft  crashes 

-43% 

-67% 

Operational  Field  Test 

For  the  final  operational  field  test,  use  of  the  courseware  was  observed  and  assessed  in  both  Army 
National  Guard  (ARNG)  and  Army  Active  Component  (AC)  aviation  units  that  represented  cargo, 
attack,  and  utility  aircraft,  missions,  and  operating  conditions.  The  field  test  employed  the 
prototype  courseware  in  a  field  setting  using  representative  groups  of  the  target  training  audience. 
Contractor  instructors  used  unit  classrooms  and  facilities  to  present  the  Aircrew  and  Instructor 
Courses  to  a  cross  section  of  31  highly  experienced  aircrew  members  and  instructors.  Data  were 
gathered  on  Aircrew  and  Instructor  Course  usability,  effectiveness,  and  areas  that  needed  interface 
and  content  improvement.  Participant  rating  results  are  shown  in  Table  6. 
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Table  6.  Operational  Field  Test  Participant  Ratings  Results 


Rating  Items 


Tutorial  provided  knowledge  to  navigate  courseware 

Courseware  structure  and  navigation  logical  and 
understandable 


Comprehension  checks  understandable 

Amount  of  information,  pace  and  time  good  (3.0  optimal) 

Mission  vignettes  allowed  observation  and  related  ACT  to  RM 

Instructor  knowledge,  preparation,  and  clarity 

Student  workstation  equipment 

Performance  Evaluation  System  and  Aircrew  Guide  helpful 
Content  relevant 


Positive  effect  on  flying  safety 

Effect  as  refresher  to  previous  Aircrew  Coordination  Training 
Positive  effect  on  mission  effectiveness 


Aircrew 

Course 

4.0 

4.0 


3.5 

3.1 

3.8 

4.4 


4.2 


3.9 


3.8 
4.0 
4.1 

3.9 


Instructor 

Course 


3.9 


3.3 

4.1 

4.5 


4.0 


Lessons  helped  understand  how  to  identify  ACT  behaviors 

Lessons  helped  understand  how  to  evaluate  ACT 
performance 

Lessons  helped  understand  how  to  facilitate  ACT  performance 
improvement 

Instructor  Evaluator  Guide  helpful  for  implementing  ACT  in 
daily  ops 


3.9 

4.2 


4.0 


4.0 


Example  operational  field  test  participant  comments  include: 

•  “The  combination  of  self-paced  computer  and  classroom  instruction,  as  well  as 
the  facilitated  discussion  helped  to  enhance  absorption  of  material.”  (Aircrew 
Course) 

•  “Working  through  scenarios  and  evaluating  the  crews  in  those  scenarios  [was 
most  helpful].”  (Instructor  Course) 

•  “It  got  you  thinking  about  ACT.  Also,  the  case  studies  were  good  but  could 
have  been  much  better  by  having  more  in-depth  detailed  study.”  (Instructor 
Course) 
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Meeting  the  Challenge 

The  NTC  rotation  was  just  what  the  unit 
had  expected.  Very  dusty  conditions  flown 
at  high  gross  weights  and  under  day  and 
night  vision  goggle  scenarios.  Under  these 
conditions,  the  unit  aircrews  applied  the 
behaviors  and  skills  they  had  re-learned 
during  their  participation  in  the  previous 
ART  ACTE  usability  assessment  program. 

They  rigorously  applied  the  crew 
coordination  objectives  (CCOs)  and  basic 
qualities  (BQs),  and  conducted  thorough, 
well-documented  after-action  reviews 
(AAR).  After  each  mission  they  ensured 
that  the  lessons  learned from  AARs  were 
disseminated  to  all  aircrew  members  in 
the  unit.  As  a  result  of  this  meticulous  application  of  ACT,  the  unit  had  a  productive  and  incident- 
free  NTC  rotation.  The  crews  were  especially  proud  of  being  able  to  apply  their  enhanced  skills  to 
trap  developing  errors  in  the  execution  phase  of  each  mission,  and  mitigate  the  risk  of  these  errors 
to  manageable  levels.  However,  the  challenge  of  maintaining  these  skills  was  still  a  significant 
concern.  The  unit  Commander  stated  that  ACT,  as  a  perishable  skill,  deteriorates  without 
command  emphasis  and  sustainment  training.  The  solution  he  suggested  was  for  the  command  to 
emphasize,  and  the  Army  to  provide  sustainment-training  products  to  ensure  that  crew 
coordination  errors  are  minimized. 

Fielding  Effective  Training  Products 

Can  interactive  multimedia  courseware  using  web-based  distribution  provide  the  realism  and 
relevance  necessary  for  effective  behavior-based  team  training  and  evaluation?  The  results  of  this 
applied  research  effort  lead  to  the  conclusion  that  the  prototype  Aircrew  Course  and  Instructor 
Course  provide  operational  situations  that  effectively  convey  ACT  information  to  multiple  mission 
aircraft  audiences.  User  testing  and  validation  results  indicate  high  levels  of  acceptance  for  both 
the  training  and  performance  evaluation  components.  Initial  testing  of  the  prototype  courseware  on 
the  Army’s  distance  learning  suite  supports  both  the  web-based  and  instructor  facilitated  delivery 
strategies  for  Army-wide  implementation. 

The  prototype  products  from  the  first  phase  of  research  include  two  interactive  multimedia  courses 
of  instruction  with  supporting  training  materials.  The  Aircrew  Course  and  Instructor  Course 
include  a  fully  integrated  Data  Management  System  that  tracks  student  demographics,  provides 
graphic  feedback  displays  during  evaluation  exercises,  and  facilitates  electronic  course  critiques. 
The  prototype  courseware  designed  and  developed  to  support  ACTE  research  used  the  collective 
experience  of  the  Aircrew  Coordination  Working  Group  and  the  contractor,  innovative  application 
of  proven  processes  and  procedures,  and  Army  and  training  industry  preferred  technology  and 
products.  ACTE  training  effectiveness  results  from  systematic  evaluation  and  testing  by  aviation 
units  in  the  field  serve  to  validate  the  application  of  experience,  innovation,  and  advanced 
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instructional  technology  in  the  development  of  this  prototype  courseware.  Furthermore,  prototype 
courseware  products  are — 

•  Authored  using  Army  and  industry  accepted  authoring  software 

•  Tested  and  demonstrated  on  the  Army’s  distance  learning  system 

•  Manageable  using  approved  Army  course  management  systems 

•  Upgradeable  to  comply  with  Shareable  Content  Object  Reference  Model 
(SCORM)  guidelines 

•  Documented  in  sufficient  detail  to  readily  meet  certification  requirements 

Sustaining  Performance  Improvements 

The  persistent  challenge  is  how  to  sustain  and  advance  the  cultural  and  team  performance 
improvements  achieved  by  initial  ACT  training  and  enhancements.  Key  elements  for  an  effective 
long-range  strategy  include  actions  to: 

•  Fully  integrate  ACT  into  the  organizational  structure,  command  climate,  rules, 
and  regulations  that  set  the  stage  for  daily  flying  operations 

•  Institute  ongoing  ACT  program  evaluation  and  sustainment  activities  to  keep 
training  realistic  and  relevant  to  operational  missions  and  conditions  (Abell, 

2000) 

•  Correlate  accident  investigation  and  accident  data  analysis  to  the  ACT  program 
structure  (behaviors)  so  that  accident  investigation  data  can  be  used  to  target 
specific  areas  of  the  ACT  training  program  for  increased  emphasis 

•  Include  instruction  on  strategies,  tools,  and  techniques  that  apply  ACT 
behaviors  and  skills  to  manage  risk  and  avoid,  trap,  or  mitigate  aircrew  error 

ACTE  Phase  II  research  is  currently  underway  and  will  further  advance  the  upgraded  program  by 
designing  and  delivering  a  comprehensive  train-the-trainer  course;  developing  training  support 
packages  for  specific  aircraft  and  missions;  integrating  ACT  into  Non-rated  Crewmember  and 
Flight  School  XXI  institutional  courses;  developing  a  prototype  evaluation-based  feedback  system; 
and  including  ACT  in  distance  learning  developments. 

Phase  III  research  will  focus  on  incorporating  the  products  of  Phase  I  and  II  as  part  of  normal 
operations  and  deploying  advanced  ACT  applications.  It  will  include  the  development  of  ACT 
event-driven  scenarios  for  multiple  aircraft  missions  and  extending  ACT  scenarios  to  advanced 
simulators  and  distributed  interactive  simulation  training  exercises. 
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Relationship  of  Aircrew  Coordination  Objectives,  Basic  Qualities,  and  Risk  Management  Process 

Crew  Coordination  Objectives  Aircrew  Coordination  Basic  Qualities  Risk  Management  Process 


Appendix  A 


A-l 


Appendix  B 


ACT  Performance  Evaluation  Checklist 
For  use  of  this  form,  see  the  ACT  Aircrew  Guide 


CCO 

BQ 

Crew  Coordination  Objectives/Basic  Qualities 

Rating 

1 

Establish  and  Maintain  Team  Relationships 

1 

Establish  and  Maintain  Team  Leadership  and  Crew  Climate 

2 

Mission  Planning  and  Rehearsal 

2 

Premission  Planning  and  Rehearsal  Accomplished 

3 

Application  of  Appropriate  Decision  Making  Techniques 

3 

Establish  and  Maintain  Workload  Levels 

4 

Prioritize  Actions  and  Distribute  Workload 

5 

Management  of  Unexpected  Events 

4 

Exchange  Mission  Information 

6 

Statements  and  Directives  Clear,  Timely,  Relevant,  Complete 
and  Verified 

7 

Maintenance  of  Situational  Awareness 

8 

Decisions  and  Actions  Communicated  and  Acknowledged 

9 

Supporting  Information  and  Actions  Sought  from  Crew 

5 

Cross-Monitor  Performance 

10 

Crewmembers  Actions  Mutually  Cross-Monitored 

11 

Supporting  Information  and  Actions  Offered  by  Crew 

12 

Advocacy  and  Assertion  Practiced 

13 

Crew/Flight  After-Action  Reviews  Accomplished 

Remarks:  (Use  continuation  sheet[s]  if  necessary) 


Notes: 

Consult  the  ACT  Aircrew  Guide  evaluation  procedures  and  guidelines.  Enter  a  summary  rating 
(1  -  7)  in  the  rating  block  for  each  ACT  Crew  Coordination  Objective  (CCO).  Refer  to  the  rating 
scale  below. 


Below 

Meets 

Exceeds 

Standard 

Standards 

Standards 

1 

2 

3 

4 

5 

6 

7 
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